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Executive Summary 


The PBS KIDS Play & Learn Science app was developed as part of the Corporation for Public 
Broadcasting (CPB) and Public Broadcasting Service (PBS) Ready To Learn Initiative, funded by the 
U.S. Department of Education. This study explored the impact of the PBS KIDS Play & Learn Science 
app, when used in a supportive context, on 


» children’s understanding of science concepts and use of science and engineering 
practices; 


» children’s use of science vocabulary; 
» child and parent-child engagement in science and engineering; and 


» parent confidence supporting their child’s science learning. 


PBS KIDS Play & Learn Science App and Supports 

The Play & Learn Science app introduces foundational science concepts and science and 
engineering practices through five distinct sets of activities: Water Games, Ramp and Roll, Shadow 
Play, Weather Control, and Gear Up. Each activity set includes three digital games and a designated 
parent page that incorporates suggestions for supporting their children’s game play (Tips) and their 


related real-world explorations (Activities). 


For this study, the research team developed a four-week structured experience to support exploration 
of the Play & Learn Science app. Families were encouraged to focus on one set of in-app and hands- 
on activities each week. Two family science nights allowed families to preview these activities and 


receive take-home supplies. 


Methods 


Participants were recruited from two child-care centers: a private child-care center in the South that 
accepted child-care fee waivers, and a Head Start center in the Northeast. Sample sizes ranged from 


31 to 33 participants by outcome measure. Children’s ages ranged from 37 to 67 months. 


This study had a single-group design with pre- and post-experience parent surveys and child 
assessments. The research team developed non-standardized child assessments for this study, 
which were tailored to Play & Learn Science content. Children participated in pre- and post- 
experience vocabulary and performance-based assessments. The multiple-choice vocabulary 


assessment measured children’s receptive English language vocabulary for key words from the Play 
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& Learn Science app. The performance-based assessment consisted of five researcher-developed 
hands-on tasks, each designed to assess focal concepts and skills from one or more Play & Learn 
Science activity sets. 


Findings 
Findings suggest that when used in a supported context, the PBS KIDS Play & Learn Science app 
can benefit both children and their parents. 


Child outcomes include 


» gains in understanding of science content and in use of science and engineering 
practices; 


» increases in use and understanding of science vocabulary; and 


» increased excitement about science, technology, engineering, and mathematics. 


Parent outcomes include 


» increased parent-child engagement in science- and engineering-related activities; and 


» increases in confidence for supporting their child’s science learning. 


Limitations 

This study’s single-group design limits our ability to make claims about causation and does not meet 
What Works Clearinghouse evidence standards (What Works Clearinghouse, 2017). In addition, we 
cannot claim that these findings would extend to average users of the Play & Learn Science app, 
because this study included additional supports based on the app. Further, the study used 
researcher-designed assessments aligned to the Play & Learn Science app content because no 
existing assessments were appropriate for this study’s content and design. We cannot say whether 


gains found in this study would translate to standard academic assessments. 


Conclusions 

As part of the CPB-PBS Ready To Learn Initiative, the PBS KIDS Play & Learn Science app is 
intended to support families in exploring science together. The findings suggest that a high-quality 
digital app can be a catalyst for real-world science exploration, especially when it incorporates tips for 
parents about how to mediate children’s use of science-focused digital games, explicit guidance for 


related real-world science activities they can do at home with their children, and specific suggestions 
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for interacting with their children in ways that stimulate science exploration, thinking, and 
conversation. Findings also suggest that parents benefit from direct guidance that helps them 
navigate the app, modeling of the hands-on activities and interaction suggestions in the app, and 
supports and encouragement for using the app in a structured and sequential manner. These results 
add to a body of research that demonstrates the potential of educational media to support children’s 
learning, particularly when adults support their children’s media use. These findings underscore the 
value of current CPB-PBS Ready To Learn family-focused science outreach programs and suggest 
that schools, child-care centers, and other organizations aimed at increasing family engagement in 


science learning should consider using the app as part of their own family science events. 
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Background 


Objective 

This report presents findings on the impact of the PBS KIDS Play & Learn Science app, when used in 
a supportive context, on children’s science learning—their early understanding of science concepts, 
their use of science and engineering practices, their use and understanding of science vocabulary, 
and their interest in doing and talking about science. The PBS KIDS Play & Learn Science app was 
developed as part of the CPB-PBS Ready To Learn Initiative, funded by the U.S. Department of 
Education. 


The importance of science education in the 21st Century 

Science literacy has become essential for navigating the demands of our increasingly science- and 
technology-oriented world and for pursuing training and careers across the workforce. Furthermore, 
our democracy depends on the ability of all citizens to make evidence-based decisions on a range of 
science-related topics and to access, understand, analyze, and evaluate information from a vast 
number of sources. It is critical that a 21st-century science education introduces students to core 
science ideas and concepts, engages them in using the science and engineering practices, cultivates 
their critical thinking, and fosters their scientific habits of mind. Furthermore, the science education 
community must commit itself to promoting equity for all students (National Research Council [NRC], 
2012). The Next Generation Science Standards (NGSS; NGSS Lead States, 2013) exemplify this 
vision. The NGSS incorporate core ideas, practices, cross-cutting concepts, and a learning 
progressions approach that recognizes the capacity of all K-12 students to do and learn science. 
However, children’s science learning begins long before they reach the kindergarten door (McClure et 
al., 2017). Additionally, increasing attention is being paid to the science that children learn outside of 
school, as well as to the roles that families, informal experiences, and digital media can play in 
supporting and fueling children’s science learning, inquiry, and interests (NRC, 2009). 


Science education should start before kindergarten 

Research clearly indicates that young children are primed to engage with the concepts and practices 
of science long before they begin formal schooling (Duschl, Schweingruber, & Shouse, 2007; 
Gelman, Brenneman, Macdonald, & Roman, 2009; NRC, 2012; National Science Teachers 
Association, 2014), and that high-quality inquiry-based science experiences in the early years lay a 
foundation for later academic learning and achievement across the developmental domains (Allen & 
Kelly, 2015; Sagkes, Trundle, Bell, & O’Connell, 2011). Doing and learning science promote 
children’s conceptual learning, their ability to grasp more abstract ideas, and their development of 


executive function and self-regulation (Kirschner, Sweller, & Clark, 2006; Weisberg, Hirsh-Pasek, & 
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Golinkoff, 2013). Cognitively challenging, personally meaningful science experiences also boost 
children’s motivation, persistence, and interest in doing and learning science, and influence their later 
school achievement and career choices (Heckman, Stixrud, & Urzua, 2006; Institute of Medicine, 
2000; Lindahl, 2007). 


Challenges to science in the early years 

Many young children do not have access to high-quality science experiences and, similar to the 
“literacy opportunity gap,” the “science opportunity gap” already exists by the time children enter 
kindergarten (American Educational Research Association, 2016). Many teachers and families of 
children ages 3 to 6 report that they lack the knowledge and confidence to support children’s early 
science learning (Gerde, Pierce, Lee, & Van Egeren, 2018; Silander et al., 2018), and that an 
emphasis on literacy in early childhood classrooms means that little time is devoted to science in 
preschool and kindergarten (Gelman et.al., 2009). Implicit biases about science, misconceptions 
about its relevance and appropriateness for young children, and a lack of understanding of its 
increasing significance at all levels of the workforce may even cause some adults to steer children 
away from science and other STEM (science, technology, engineering, and mathematics) 
opportunities (Bian, Leslie, & Cimpian, 2017; Kadlec, Friedman, & Ott, 2007; Newitz, 2014). 


The critical role of parents and families 

The 21st-century demand for a more science-literate population and workforce has provoked a 
national dialogue about science education in schools, including for our youngest students. However, 
current research indicates that young children’s informal science experiences may be even more 
influential than school science experiences in sparking their motivation to do and learn science 
(Crowley et al., 2001; Turkle, 2008) and that families are powerful mediators of their children’s 
science activities, interests, and learning (Silander et al., 2018; Turkle, 2008; Weiss, Little, Bouffard, 
Deschenes, & Malone, 2009). Informal experiences are compelling because they enable children to 
pursue their individual science interests, address children’s need to figure things out and solve 
problems, and forge connections between the science children are learning and real-world 
applications (Crowley et al., 2001). Specific behaviors of parents that are associated with sustaining 
children’s interests in science activities are having fun themselves, getting actively involved, initiating 
and sustaining science talk, and making connections to their children’s daily lives (Fender & Crowley, 
2007; Liebham, Alexander, & Johnson, 2013; Rigney & Callanan, 2011). A growing body of research 
points to the importance of parent-child conversations for igniting children’s science dispositions and 
supporting their causal and conceptual reasoning (Callanan, Castaneda, Luce, & Martin, 2017; 
Haden, 2010). However, science can be a challenging domain for parents to define and support, and 
many parents lack the confidence, information, and resources they need to nurture their children’s 
natural scientific impulses (Silander et al., 2018). 


Ready To Learn Research 7 


Dn 

PBS KIDS Play & Learn Science Evaluation Report 
The potential of digital media to support children and families 
The Public Broadcasting Service (PBS), in partnership with the Corporation for Public Broadcasting 
(CPB), has developed a range of digital resources to support children’s and families’ science learning. 
These resources were developed as part of the CPB-PBS Ready To Learn Initiative, funded by the 
U.S. Department of Education, to promote school readiness, particularly for low-income children. 
Studies of Ready To Learn resources have shown that digital media can further children’s learning, 
particularly when knowledgeable and nurturing adults mediate their use by coviewing, engaging, and 
interacting with children as they view and play (Anderson, Huston, Schmitt, Linebarger, & Wright, 
2001; Fisch, Truglio, & Cole, 1999; Hurwitz, 2018; Moorthy et al., 2013; Pasnik, 2018; Rideout, 2017; 
Schmidt & Anderson, 2006). The National Association for the Education of Young Children (NAEYC) 
and the Fred Rogers Center (2012) agree that technology and media can support positive family/child 
relationships, as well as learning, when adults mediate their use. CPB and PBS, as leaders in 
educational media, are likewise committed to creating and disseminating digital resources that 
support 2- to 8-year-old children’s foundational science learning in relation to the NGSS and that 
enable parents to more fully realize their potential in promoting their children’s science learning. 
However, families in a previous study (Silander et al., 2018) reported that their children were less 
likely to learn science or math from educational media—in comparison to literacy, for example—and 
that they, as parents, were less likely to help their children make connections between science 
concepts introduced in media and their daily family activities. Parents also said that they would do 
more science with their children if they had access to engaging, high-quality science content, ideas 
for activities they could do at home with readily accessible and inexpensive materials, information 
about what their children should be learning, and tips about how they can best support science 


learning (Silander et al., 2018). 


PBS KIDS Play & Learn Science App 

To address this need, the CPB-PBS Ready To Learn Initiative developed the PBS KIDS Play & Learn 
Science app. This app (described below) is intended to support family science exploration both within 
the app and through direct hands-on explorations for children ages 3 to 5 and their parents. High- 
quality media can help adults broaden their understanding of the nature of science, enabling them to 
view it as an active, collaborative, and creative process of questioning, investigating, sharing 
observations and findings, and figuring out how the world works (Goldstein, Christensen, Gerard, & 
Silander, 2018). Media resources also can provide parents with ideas for at-home science 
explorations that are engaging to their children, accessible and affordable, and incorporate explicit 
guidance for asking questions that fuel children’s science inquiry, thinking, and motivation (Silander et 
al., 2018; Troseth, Saylor, & Archer, 2006). The independent research organizations Education 
Development Center (EDC) and SRI International (SRI) explored how well the PBS KIDS Play & 


Learn Science app and associated supports engage children and families in science practices and 
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concepts. We implemented this experience in child-care centers in two U.S. sites, one in a small city 


in the South, and the other in a small city in the Northeast. 


PBS KIDS Play & Learn Science App and 
Supports 


Theory of change 

The objective of this study was to evaluate the impact of the Play & Learn Science app, when used in 
a supportive context, on children’s science learning—their early understanding of science concepts, 
their use of science and engineering practices, their science vocabulary, and their interest in 
exploring and talking about science topics. The research team developed a structured study 
experience intended to support family engagement and joint parent/child media use. To optimally and 
intentionally promote their children’s science learning, parents need guidance about appropriate 
content, activity ideas for supporting their children’s inquiry and exploration during their families’ daily 
lives, and explicit tips for interacting with their children in ways that promote science exploration and 
that help them think and talk about their experiences, observations, and ideas (Silander et al., 2018). 
Such guidance is provided in the Play & Learn Science app itself, and this study further focused on 
support for parents by emphasizing a limited number of games and related activities, creating a 
specific structure and sequence for what topics and activities families would explore, providing 
materials and books that elaborate on target concepts and practices, and modeling the activities, tips, 
and conversational prompts during face-to-face family science nights. An additional support for 
families was a guide that provided information about the study and the app and reinforced the 
suggested activities and tips available in the app. Other resources were a tablet, two face-to-face 
family science nights, a variety of simple, everyday exploration materials to take home, and reminder 


texts to use the app. 


PBS KIDS Play & Learn Science app 

The Play & Learn Science app introduces basic science concepts and science and engineering 
practices by way of five distinct sets of in-app and hands-on activities: Water Games, Ramp and Roll, 
Shadow Play, Weather Control, and Gear Up. Each set of activities includes three digital games and 


a designated parent page that incorporates the following: 
» Tips. Information for parents about the concepts and skills the game introduces, as well 


as suggestions for how they can interact with children in ways that support their science 
inquiry and learning and help them make connections to everyday life experiences. 
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» Activities. Three or four “real-world” hands-on activities that promote further exploration 
of the concepts introduced in the games; integrate use of the science and engineering 
practices; and include specific implementation/facilitation suggestions for parents. One 
activity in each set incorporates a list of suggested children’s books that center on the 
relevant target concepts. 


» Journal. Images from the target games with explicit suggestions for comments and 
questions parents can use to support their children’s reflection on their game 
experiences. 


Water Games and Ramp and Roll introduce physical science concepts related to the NGSS core 
ideas (1) Matter and Its Interactions and (2) Motion and Stability: Forces and Interactions. Shadow 
Play introduces concepts related to the core idea Waves and Their Applications in Technologies for 
Information Transfer. Weather Control and Gear Up introduce Earth and Space Science concepts 
related to the core idea Earth’s Systems. The PBS KIDS Science Learning Framework’ describes 
what 2- to 8-year-old children might be expected to do and learn in relation to these core ideas, 
concepts, and practices. All the activity sets, in the combined context of the games and the real-world 
explorations, engage children in using science and engineering practices with a focus on planning 
and carrying out investigations, analyzing and interpreting data, using mathematics and 
computational thinking, constructing explanations, and obtaining, evaluating, and communicating 


information. 


Each week the study focused on one activity set. The research team suggested games and activities 
within that set for parents to play with their children (Exhibit 1). We selected games and activities 
based on the opportunities they provided for children’s active engagement, how well the activities 
related to the target games, how easily parents could implement and facilitate the activities at home, 
and our own capacity to implement and model the activities at the face-to-face family science nights. 
The components of the intervention in this study are detailed in Exhibit 1. 


' Development of the PBS KIDS Science Learning Framework has been informed by the Next Generation Science Standards and other 
learning standards, and guided by early science advisors as part of the CPB-PBS Ready To Learn Initiative 
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Exhibit 1. Study schedule of PBS KIDS Play & Learn Science app activities and supports 


Activity set 


Week 1: 
Ramp and Roll 


Week 2: 


Water Games 


Week 3: 
Shadow Play 


Week 4: 
Gear Up and 
Weather Control 


Ready To Learn Research 


sk-hge (ime fchaat= 


Explore the Roll 


Supplement: Hit the 
Target 


Sink or Float 


Guess the Shadow 
Supplement: 


Exploring Shadows 


Gear Up 
Supplements: 


The Amazing 
Umbrella 


Thermometer Picking 


Related parent activity 


Roll, Slide, or ... Not! 


Supplement: How Far 
Will It Go? 


Does It Sink or Float? 


Flashlight Shadows 
Supplements: 
Changing Shadows 


Animal Shadow 
Puppets 


Gear Up 

Let’s Go Outside 
Supplements: 

The Amazing Umbrella 


Testing materials with 


water 


Thermometer activity 


Take-home family science 


materials 


Cove molding ramp 


Bag of objects for 


rolling/sliding 
Target 


Book about motion 


Sink/float prediction chart 


Bag of objects for 
sinking/floating 


Book about sinking/floating 


Flashlight 


Materials to make shadow 


puppets 


Sidewalk chalk for tracing 


shadows 


Book about shadows 


Weather chart 

Plastic pipettes 

Four different materials 
Child-safe thermometer 


Book about weather 
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Family science nights 

The study included two family science nights, one at the beginning and one midway through the 
study. Attendance at the first science night was mandatory for families to be part of the study. At that 
session, parents completed a pre-experience survey and received a tablet computer they could keep. 
Attending the second science night was optional; however, attendance at both family nights was 


monetarily incentivized. 


The research team drafted plans for both family science nights, incorporating feedback from staff at 
the centers participating in the study and educators at WCTE?. The activities presented at each night 
modeled the games and activities that families would be asked to do at home during the subsequent 
two weeks. 


The research team and local center staff facilitated the family science nights, which lasted 1.5 to 2 
hours. The science nights began with a shared meal and a brief discussion of the evening’s focal 
science content, including tips for engaging young children in science and an introduction to the 
evening's activities. Families were provided with “passports” to support their navigation through three 
to four facilitated stations and were invited to rotate among the stations for the rest of the night. At the 
first family night, the station topics were Ramp and Rolland Water Games. At the second family 
night, the stations focused on Shadow Play and Gear Up/Weather Control. Each station had 
materials and tools for two to four investigations that mirrored the parent activities in the app and 
signage with recommended conversation prompts to stimulate parent-child conversation about each 
activity. Each station also offered an opportunity to record predictions or findings for at least one of 
the activities. 


At both family nights, children and their parents had an opportunity to become familiar with the app 
and to play the target games. At the first family night, the app was introduced at a separate station. At 
the second family night, use of the app was integrated into each station so that families could make 
closer connections between the games, tips, and activities in each activity set and the related hands- 
on investigations. Facilitators supported families’ use of the app and all hands-on investigations, 
encouraging parent-child exploration and conversation. Facilitation of the app included familiarizing 
children and their parents with the games, how to navigate through them, and how to locate the family 


activities and tips within the app. 


?WCTE is a PBS member station that works closely with the CPB-PBS Ready To Learn Initiative and supports local outreach 
programming. 
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Additional intervention supports 
In addition to the family science nights, this study supported families’ use of the Play & Learn Science 


app at home by providing the following: 


» family guide with instructions for using the tablet, the Play & Learn Science app, and 
each focal parent activity; 


» calendar of study events and weekly topics; 


» take-home materials for family activities to do at home and demonstrated during the 
family science nights (see Exhibit 1); and 


» eight text messages reminding parents of study activities, as well as of weekly topic 
areas and associated games and parent activities. 


Because all these resources and materials were used at home, we could not support families’ use of 
them directly. However, facilitators kicked off the second family science night with a reflection, inviting 
children and families to describe what activities they had done together at home and asking parents 
to share what they thought about the activities and how their children responded to them or extended 


Study Design and Methods 


Research questions and study design 


This study pursued five research questions. 


» What is the change in children’s understanding of science concepts and use of science 
and engineering practices after four weeks of exposure to the PBS KIDS Play & Learn 
Science app and supports? 


» What is the change in children’s use of target vocabulary after four weeks of exposure to 
the PBS KIDS Play & Learn Science app and supports? 


» To what extent do parents report that four weeks of exposure to the PBS KIDS Play & 
Learn Science app and supports improved their children’s science and/or engineering 
concepts, skills, and vocabulary? 


» To what extent do parents report that four weeks of exposure to the PBS KIDS Play & 
Learn Science app and supports promoted children and their parents doing and talking 
about science/engineering? 
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» To what extent do parents report that four weeks of exposure to the PBS KIDS Play & 
Learn Science app and supports improved their confidence in or ability to support their 
child’s science/engineering learning? 


To answer these questions, this study had a single-group design with pre- and post-experience child 


assessments and parent surveys. 


Recruitment and incentives 
This study targeted child-care centers serving low-income families who had not participated in PBS 
KIDS science outreach in the past. Site 1 was a private child-care center in the South that accepted 


child-care fee waivers. Site 2 was a Head Start center in the Northeast. 


Centers distributed recruitment flyers and registration and consent forms to families. Each site was 
encouraged to recruit 15 to 20 families who were likely to meet the eligibility criteria. To be eligible, 


families had to 


» agree to attend the first family science night so they could learn about the study and 
receive their tablet; 


» have at least one child aged 3- to 5-years old; and 


» be fluent in English (both parent® and child). 


Recruitment did not target low-income families specifically. Upon receipt of the registration forms, 


researchers called families to screen for eligibility. 


Participants received $50 for participating in the first family science night; $50 for participating in the 


second family science night; and $50 for completing the parent post-experience survey. 


Child assessments 

The research team developed non-standardized child assessments for this study, which were tailored 
to the Play & Learn Science content. The children participated in pre- and post-experience vocabulary 
and performance-based assessments that the research team administered during the school day. 
Each assessment lasted 20-25 minutes. Pre-experience child assessments were administered on the 


day of or the day before the first family science night; all these assessments were completed before 


3 For simplicity, we refer to adult participants in this sample as parents, although the sample may have included a range of caregivers 
such as grandparents and legal guardians. 
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the start of the first family science night. Post-experience child assessments were administered after 


the four-week intervention period. 


The multiple-choice vocabulary assessment measured children’s receptive English language 
vocabulary for key words from the Play & Learn Science app. It consisted of two training items and 19 
assessment items. Children were asked to identify which of four pictures represented the word 
spoken by the examiner. Items were arranged in ascending order of difficulty based on age- 
standardized vocabulary lists (Dale & Chall, 1948; Francis & Kuéera, 1964; Marvin, Beukelman, & 
Bilyeu, 1994). To reduce frustration, the assessor discontinued the assessment after a child 


answered five consecutive items incorrectly, scoring all subsequent items as zero. 


The performance-based assessment consisted of the following five researcher-developed hands-on 
tasks, each designed to assess focal concepts and skills from one or more Play & Learn Science 
games and their associated family materials. 


» Activity set: Ramp and Roll; focal game: Explore the Roll. Participating children were 
asked to observe and describe six objects—two rollers, two sliders, and two objects that 
can either roll or slide depending on how they are positioned—and to test how each 
object moves on a ramp. They were given a chart to sort the objects into one of three 
categories (rolls, slides, can roll and slide). For objects the children identified as rolling or 
sliding, they were asked to explain their thinking about why these objects move the way 
they do. For objects the children identified as both rolling and sliding, they were asked to 
describe or demonstrate how they could make the object roll or slide. 


» Activity set: Water Games; focal game: Sink or Float. Participating children were 
asked to observe and describe six objects—three sinkers and three floaters. They were 
asked to define the terms sink and float and to record their predictions about whether 
each object sinks in or floats on water. For two correctly sorted objects—one sinker and 
one floater—the children were asked to explain their thinking about why that object would 
sink or float, and to describe or demonstrate how they would test the objects if a 
container of water were present. 


» Activity set: Shadow Play; focal game: Guess the Shadow. Participating children 
were asked to identify which one of four objects created a shadow depicted in a 
photograph and to provide evidence for their claim. The children were then provided with 
a flashlight and asked to use the object to replicate the shadow in the photograph, 
demonstrating how they would make the shadow larger and smaller. In the second part 
of the task, children were shown four different shadows and asked to match an object to 
its shadow. If they chose correctly, they were asked to describe how the object’s shadow 
is similar to and different from the object itself. 
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» Activity set: Gear Up; focal game: Gear Up. Participating children were asked to view 
an image of a child dressed in rain gear, a clear sunny day visible through a window, and 
an outdoor thermometer indicating hot weather. Children were asked if they noticed 
anything silly about the image, and a follow-up question about the weather was asked if 
needed. Children were then asked to name the thermometer, describe what it tells us 
about the weather, and identify more weather-appropriate clothing for the child, giving a 
reason for their choices. 


» Activity set: Gear Up: focal game: The Amazing Umbrella. Participating children were 
asked to observe and describe five objects made of different materials. They were then 
asked to choose which object a toy character should use to stay dry in the rain and to 
give reasons for choosing or not choosing each object. 


Appendix A maps each task to PBS KIDS science content standards. 


Parent surveys 

Parents completed pre- and post-experience surveys. Pre-experience surveys were completed at the 
start of the first family science night. Four weeks later, parents received a text message link to 
complete the post-experience survey on their own devices. To allow for pre-post comparisons, both 
surveys asked parents about their child’s past-month science activities, excitement, and vocabulary 
usage, and about parents’ confidence in supporting science learning and joint science activities with 
their child. In addition, the pre-experience survey collected demographic information, and the post- 
experience survey collected information about perceived learning gains resulting from the Play & 


Learn Science app and supports. 


Qualitative evidence 
The second family science night began with small-group discussions of families’ science experiences 
since the first family night. We use parent quotes, taken from facilitator notes, to illustrate quantitative 


findings. These discussions were not recorded or transcribed; parent quotes may not be verbatim. 


Analysis 
We report descriptive statistics for the parent surveys. To compare pre- and post-experience scores 
on the vocabulary and performance-based tasks, we used repeated measures general linear models. 


Appendix B details how the child assessments and parent surveys were scored. 
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sample Description 


Sample size and attrition 
Exhibit 2 displays the sample size for each component of the study. Attrition was minimal. The 
remainder of this section describes the sample (N = 32) based on pre-experience survey data. 


Exhibit 2. Study sample size 


Study component Site 1 Site 2 Total 


Attended first family science night and took parent pre- re eye 32 


experience survey 


Attended midpoint family science night 16 15 31 
Took parent post-experience survey 17 16 33 
Child participated in pre- and post-experience assessment? 15° 16 31 


* One parent is represented twice in the parent survey sample. This parent enrolled two eligible children in the study and took 
each survey once for each child. We included both surveys when describing child participants and only one survey for this 
family when describing parent participants. 

> For one of 16 consented parents in Site 2, the pre-experience survey was lost due to a technical glitch. That parent was 
excluded from the parent pre- and post-experience survey sample but retained in the post-experience-only survey sample. 

° This excludes two children in Site 2 who took the baseline assessment but were ineligible because they did not attend the first 
family science night. Two eligible children in Site 1 refused the baseline assessment. These two children were excluded from 
the child assessment sample and not assessed post-experience. They participated in the study intervention. Their parents 
were retained in the survey sample. 

4 The number of children with usable data varied by child assessment. For the performance-based tasks, N = 26. For the 


vocabulary assessment, N= 31. 


Child gender, IEP status, and age 

The sample comprised 13 male and 18 female children. Two children were identified as having an 
individualized education plan or receiving special education services, one at each site. Children’s 
ages ranged from 37 to 67 months, with a mean of 51 months (SD = 10). Site 1 participants were 47 
to 66 months old. Site 2 participants were 37 to 67 months old. 
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Race/ethnicity and language 

Race/ethnicity and language differed by site. Children in the Site 1 sample were predominantly 
identified as White (non-Hispanic) with some Black/African American children, whereas the Site 2 
sample included a mix of children identified as Asian, Black or African American, and White (non- 


Hispanic), among other ethnicities (Exhibit 3). 


Exhibit 3. Child race/ethnicity 


15 
@ American Indian or Alaskan 
13 : 
Native 
Asian 
10 : : 
@ Black or African American 
8 
3 
‘e @ Hispanic or Latino 
O 


@ Native Hawaiian or Other Pacific 


5 
4 4 
Islander 
1 1 1 mf White (Non-Hispanic) 
: = os oe 


@ Other 
Site 1 Site 2 


Note: N = 32 children. Respondents were asked to select all that apply. 


English was the only language spoken at home for most children in the Site 1 sample, whereas a 
language other than English was spoken at home for most children in the Site 2 sample (Exhbiit 4). 
Parent-reported languages spoken at home in Site 2 were Arabic, Berber, Chinese (Mandarin, 
Cantonese, or other), Creole, Esan, Pakhtu, and Portuguese. For the three Site 1 children whose 
households spoke a language other than English, parents did not identify the languages. 


Ready To Learn Research 


PBS KIDS Play & Learn Science Evaluation Report 


Exhibit 4. Language spoken at home 


20 
ai ' 
3) = Only English spoken at home 
<2 10 
5 5 m Language other than English 
spoken at home 
0 


Site 1 Site 2 


Note: N = 32 children. 


Parent characteristics 

Among the participants who completed the pre-experience survey, 29 were female and two were 
male. The sample included two grandparents, one at each site. Whereas parent survey respondents 
at Site 2, the Head Start center, all reported annual household incomes of less than $50,000, those at 
Site 1, the private child-care center, reported a broader range of incomes (Exhibit 5). Parent survey 


respondents at both sites reported a range of education levels (Exhibit 6). 


Exhibit 5. Annual household income 


10 9 
m <$25,000 
2 5 ™ $25,000 to <$50,000 
© 5 
8 ™ $50,000 to <$75,000 


5 
4 
3 
3 @ S75,000 to <$100,000 
1 @ $100,000 or more 
= qa 
0 


Site 1 Site 2 


Note: N= 31 parents. 
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Exhibit 6. Parental education 


> 
4 
3 


Site 1 Site 2 


Parents 
uw 


0 


Note: N = 31 parents. 
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This section presents findings from child assessment and parent survey data. For child assessment 
data, we report the results of statistical significance tests and associated effect sizes. For parent 


survey data, we report descriptive statistics and describe apparent numerical differences. 


What is the change in children’s understanding of science concepts and use of 
science and engineering practices after four weeks of exposure to the PBS KIDS 


Play & Learn Science app and supports? 


CHILDREN DEMONSTRATED GAINS ON A MEASURE OF SCIENCE 
Pp CONCEPTS AND SCIENCE AND ENGINEERING PRACTICES. 


Total scores on the performance-based tasks designed to measure concepts and practices related to 
Play & Learn Science content showed a statistically significant increase from pre-experience 


assessment to post-experience assessment (Exhibit 7).4 The size of this effect was medium.® 


Exhibit 7. Performance-based task scores at pre- and post-experience assessment 


70% 
60% 
50% 
40% 
30% 
20% 
10% 

0% 


37% 


Percent Correct 


Pre Post 


Note: N = 26 children. Five children were removed from this analysis because they were unresponsive for most of the 


assessment. Error bars represent standard error. 


4 A repeated-measures general linear model was used to analyze the effects of time point (pre- or post-experience) on children’s 
performance-based task scores. Performance-based task scores increased significantly from pre-experience assessment to post- 
experience assessment, F(1, 25) = 29.91, p< .001. This analysis did not control for child demographic characteristics because children 
served as their own controls in this design. 

5 Cohen's d= .62. 
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Exhibit 8 presents excerpts from children’s responses to selected performance-based assessment 
items. The table presents the same child’s responses at pre- and post-experience assessments to 
illustrate within-child growth in science understanding. 


Exhibit 8. Selected responses to performance-based assessment items 


Performance-based Pre-experience Post-experience 

assessment item response excerpt response excerpt 
What do you notice about [this "Racing car is for "Can roll because it have wheels, 
car]? playing." wheels can roll." 
Why do you think [the "Because it does." "It's heavy and big and made of steel. 
marble] would sink? It would sink." 
Why didn't you choose [felt] to "Because it's soft." "Because it's soft, the rain will make it 
keep Woody dry? wet; the rain will get past it." 


Note: This table displays responses from three children. For each performance-based assessment item, the table displays the 


same child’s pre- and post-experience responses to show within-child growth over time. 


What is the change in children’s use of target vocabulary after four weeks of exposure to the 


PBS KIDS Play & Learn Science app and supports? 


CHILDREN SHOWED GAINS ON TWO CONTENT-SPECIFIC MEASURES OF 
Pp VOCABULARY. 


Children’s scores on the multiple-choice vocabulary assessment were statistically significantly higher 
post-experience than pre-experience (Exhibit 9).° The size of this effect was small.” Appendix C 
presents descriptive statistics for all words included on the multiple-choice, performance-based, and 
parent-reported vocabulary measures, and Exhibit 8, above, presents examples of children’s 
vocabulary use. 


5 A repeated-measures general linear model was used to analyze the effects of time point (pre- or post-experience) on children’s 
multiple-choice vocabulary assessment scores. Scores at post-experience assessment were significantly higher than scores pre- 
experience assessment, F(1, 30) = 16.39, p< .001. 

7 Cohen's d= .45. 
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Exhibit 9. Vocabulary assessment scores at pre- and post-experience assessment 


70% 63% 
60% 
50% 
40% 
30% 
20% 
10% 

0% 


Percent Correct 


Pre Post 


Note: N = 31 children. Error bars represent standard error. 


There was a statistically significant increase in children’s use of vocabulary words during the 
performance-based task administered at the end of the study, compared to the same task at the 
beginning of the study (Exhibit 10).8 This effect size was small to medium.° 


Exhibit 10. Vocabulary word usage at pre- and post-experience assessment 


_ 30 
& 
E 25 19.2 
E B 20 
> 
> 
a> | 
Oo Oo 
5 5 
“ 0 


Pre Post 


Note: N = 28. Three children were excluded from this analysis because no utterances were recorded during the performance- 


based tasks at pre-experience assessment, post-experience assessment, or both. Error bars represent standard error. 


8 A repeated-measures general linear model was used to analyze the effects of time point (pre- or post-experience) on children’s 
vocabulary usage counts. Vocabulary usage was significantly greater at post-experience assessment than at pre-experience 
assessment, F(1, 27) = 13.121, p< .05. 

° Cohen's d= .47. 
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To what extent do parents report that four weeks of exposure to the PBS KIDS Play & Learn 
Science app and supports improved their children’s science and/or engineering concepts, 


skills, and vocabulary? 


PARENTS REPORTED THAT THE PBS KIDS PLAY & LEARN SCIENCE APP 

Pp AND SUPPORTS HAD A POSITIVE IMPACT ON THEIR CHILD’S SKILLS IN 
AND EXCITEMENT ABOUT STEM AND ON THEIR USE OF TARGET 
VOCABULARY. 


Nearly all—32 of 33—parents reported that their child learned something new from Play & Learn 
Science games or hands-on activities. Of those 32 parents, 30 indicated that their child learned 
something new about ramps, rolling, or sliding; sinking and floating; and shadows. Twenty-six 
reported that their child learned something new about weather. Almost all parents reported that the 
Play & Learn Science games and activities had a somewhat positive or extremely positive impact on 
their child’s STEM and literacy skills and interests (Exhibit 11) and their excitement about STEM 
subjects (Exhibit 12). Parents reported the smallest pre- to post-experience survey gains for their 
child’s excitement about literacy, most likely because literacy was not a focus of the Play & Learn 
Science app and supports. 


Exhibit 11. Parent-reported impact on child's skill/knowledge domains 


m@ Extremely positive impact m Somewhat positive impact 
@ Neither positive nor negative impact & Somewhat negative impact 


@ Extremely negative impact 


35 4 
30 
é 25 
= 20 
= 415 
10 
5 
0 
Science Technology Engineering Math Using tools like Reading, 
measuring writing, or 
tools, literacy 
magnifiers, or 
thermometers 


Note: N = 33 children. 
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Exhibit 12. Parent-reported child excitement over the past month 


m Pre m Post 
5 4A 4.3 
41 4.0 ; 4.0 42 

4 37 3.7 
i 3.3 EBS) 3.4 
E 3 22 
= 
oO 
x< 
7 Hl 

1 

Science Technology Engineering Math Using tools like Reading, writing, 


measuring tools, or literacy 
magnifiers, or 
thermometers 


Note: N = 32 children for all items except reading, writing, or literacy (N = 31); and engineering (N = 31). One parent did not 
respond to these items on the pre-experience survey. For comparability, this parent’s post-experience survey responses for 


these items are omitted. 


The survey asked parents to select any words they heard their child say or try to say in the last month 
from a list of 36 target vocabulary words from the Play & Learn Science app (see Appendix C). The 
number of selected words increased from the pre-experience survey to the post-experience survey 
(Exhibit 13) 1° 


Exhibit 13. Parent-reported child past-month vocabulary word usage 


30 28.4 
ae 23.6 


Pre Post 


Note: N = 32 children. 


1° We did not test whether this increase is statistically significant. 
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In a discussion during the second family science night, one parent described the change in a child’s 
vocabulary: “She is playing with her toys in the tub like she always does, but now she is using words 


like ‘sink’ and ‘float’ that | haven’t heard her use before.” 


To what extent do parents report that four weeks of exposure to the PBS KIDS Play & Learn 
Science app and supports promoted children and their parents doing and talking about 


science/engineering? 


PARENTS REPORTED AN INCREASE IN ENGAGING IN CHILD AND 
Pp PARENT-CHILD SCIENCE ACTIVITIES. 


Children doing science 


The pre- and post-experience surveys asked parents how many times their child did or talked about 
activities related to the Play & Learn Science content. Parent-reported frequency of these activities 
was higher on the post-experience survey than on the pre-experience survey (Exhibit 14).'' Parent 
survey responses also indicated a pre- to post-experience survey increase in how frequently children 
engaged in more general science-related activities not specific to the Play & Learn Science app and 
supports (Exhibit 15).'2 


Exhibit 14. Parent-reported frequency of Play & Learn Science-related activities 


How many times has your child done or talked about each of the following in the last 
month? 


m@ Pre m Post 


4 3.6 3.4 3.4 92-6 

3 2.5 : 
2.0 

2 

: i 

0 


Played with objects Played with objects on Explored the shadows Talked about what to Explored things that 
that sink or float in a slide or ramp different things make wear for cold, warm, might keep you dry in 
water or rainy weather the rain 


Times 


Note: N = 32, except for “Explored the shadows different things make” (N = 31). One parent did not respond to this item on the 


pre-experience survey; that parent’s post-experience survey response was omitted for consistency. 


1! We did not test whether this increase is statistically significant. 
12 We did not test whether this increase is statistically significant. 
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Exhibit 15. Parent-reported frequency of general science-related activities 


How many times has your child done or talked about each of the following in the last 


month? 
m@Pre m Post 
+ 33 3.3 
2.9 

3 2.6 
zh a4 24 
o 
€ 2 
= 

1 

0 

Played with a science-related puzzle, Talked about science jobs or Watched TV shows or played online 
board game, or toy pretended to be a scientist, doctor, games or apps about science 


computer programmer, astronaut, 
engineer, or zookeeper 


Note: N = 32 children. 


In discussions during the second family night, one parent described how the Play & Learn Science 
app and supports encouraged her child to talk: “[My child] has some speech issues. | love the way 
these games and activities got her so excited to talk about what she was doing.” Other parents 
reported that their children engaged in science activities related to the Play & Learn Science app and 
supports. One parent said, “As soon as [my child] got in the bathtub, he started filling things up with 
water and testing things to see if they would float or sink.” Another parent described a child’s 
explorations with ramps: “[My child] tested the slides at the park to see which ones he would go faster 
and slower on. He figured out that he went faster on the big kid slide because it was steeper than the 
baby slide.” 


Parents/caregivers and children doing science 


The frequency with which parents reported engaging in science- and engineering-related activities 
with their children increased from the pre- to post-experience survey (Exhibit 16).'% 


18 We did not test whether this difference is statistically significant. 
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Exhibit 16. Parent-reported frequency of parent-child science activities 


How many times have you done the following activities with your child in the last month? 


m@Pre m Post 
4 32 34 3.4 3.5 33 
3 24 Z- 2.6 
- 2.2 
25 i 
= 
d. 
0 
Built something with Explored science in Read about nature, Talked about science Watched TV shows or 
your child (such asa _ everyday activities with science, or engineering with your childor — played online games or 
tower with blocks, a your child in books or magazines answered your child's apps about science 
model airplane, etc.) questions about science 


Note: N = 32 children for all items except “Talked about science with your child...” (N = 31). One parent did not respond to this 


item on the pre-experience survey; this parent’s post-experience survey response was omitted for consistency. 


At the second family science night, parents described engaging in science activities related to Play & 
Learn Science with their child. One parent reported engaging in both the digital and hands-on 
versions of an activity: “We played sink/float on the tablet and at bath time. We like to guess. We ask 
each other and guess.” Another parent and child described exploring ramps together. Their 
discussion illustrates parent-child talk about the family science experience. 


Parent: “We set up a ramp on the living room rug and placed the target at the end. What 
happened?” 


Child: “The ball rolled so so so fast and hit the target.” 


Parental mediation of Play & Learn Science app usage 


On the post-experience survey, most parents indicated that they frequently mediated their child’s use 
of the Play & Learn Science app in a variety of ways, including complimenting or encouraging their 
child’s use of the app, playing the app together, and talking about connections between the app and 
daily life (Exhibit 17). In addition, of 33 parents who took the post-experience survey, 32 reported that 
their child talked with them about Play & Learn Science games or hands-on activities either a little or 
a lot. 
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Exhibit 17: Parent-reported past-month mediation behaviors 


35 


30 


Parents 


In the past month, how often did you 


H4ormoretimes ®3times &2times i1time mDidnotdo 


Compliment or Explain or talk Help your child Monitor your child’s Play the Play & Talk about 
encourage your aboutsomething access the Play & playing Learn Science app connections 
child while he/she your child was Learn Science app with your child between the Play & 
used the Play & playing in the Play & Learn Science app 
Learn Science app Learn Science app and things you do in 
your daily life 


Note: N = 33 children. 


To what extent do parents report that four weeks of exposure to the PBS KIDS 
Play& Learn Science app and supports improved their confidence in or ability to 


support their child’s science/engineering learning? 


PARENTS REPORTED INCREASES IN CONFIDENCE IN SUPPORTING THEIR 
CHILD’S SCIENCE LEARNING. 


Parents reported pre- to post-experience increases in their confidence supporting science skills 
(Exhibit 18). These increases were modest, but larger than parent-reported increases in confidence 
supporting other skills. Similarly, modest pre- to post-experience increases were reported for parents’ 
past-month confidence helping their child with a new science or engineering game or activity (Exhibit 
19).'4 


14 We did not test whether these increases were statistically significant. 
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Exhibit 18: Parent confidence by domain 


How confident do you feel about helping your child learn the following skills at home? 


m@ Pre m@ Post 
3 26 2.7 2.7 
26 25. 25 2.6 
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Behavioral, social and Math skills Reading and writing Science skills 
emotional skills skills 


Note: N = 31 parents for all items except math skills (N = 30). One parent did not respond to this item on the pre-experience 


survey; their post-experience survey response was omitted for consistency. 
Exhibit 19. Parent confidence supporting a new science/engineering activity 


In the last month, how confident have you felt helping 
your child with a new game or activity about science or 
engineering? 


2.4 


Confidence 
NO 


Pre Post 


Note: N = 26 parents. Pre- and post-experience survey responses are omitted for parents who responded that this “did not 


come up in the past month.” 


At the second family science night, one parent told facilitators that the experience had increased her 
science knowledge, stating, “I’m learning too.” Another family said that before participating in the 
experience they did not know they could engage in science explorations with household objects. 
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Discussion 


As part of the CPB-PBS Ready To Learn Initiative, the PBS KIDS Play & Learn Science app is 
intended to support families in exploring, thinking, and talking about science together. The app is one 
of several PBS KIDS media properties and outreach programs designed to promote family science 
engagement, funded by a Ready To Learn grant from the U.S. Department of Education. Findings 
suggest that when used in a supported context, the Play & Learn Science app can benefit both 
children and their parents. These results add to a body of research that demonstrates the potential of 
educational media to support children’s learning, particularly when adults support their children’s 


media use. 


Child outcomes 

Child assessment and parent survey data indicated gains in children’s understanding of science 
content and practices after four weeks of exposure to the Play & Learn Science app and supports. 
We found a statistically significant pre-experience to post-experience gain in children’s scores on 
performance-based tasks aligned to Play & Learn Science content. Parents reported that their child 
learned something new about the topics covered in the app and that the app positively affected their 
child’s STEM skills. 


Three data sources indicated pre- to post-experience increases in children’s vocabulary related to 
app content. From pre- to post-experience, there was a Statistically significant increase in children’s 
scores on the vocabulary assessment and in their use of vocabulary terms during the performance- 
based assessment. Parent survey responses indicated an increase in the number of app-related 


vocabulary terms used over the past month. 


Parents also reported that the Play & Learn Science app positively affected their child’s excitement 
about STEM. Further, parent survey responses indicated that children did or talked about both app- 


specific and more general science activities more frequently during the study than prior to the study. 


Parent outcomes 

Parents reported that they engaged in science- and engineering-related activities with their children 
more frequently during the study than prior to the study. Most parents also reported that they 
mediated their child’s use of the Play & Learn Science app. Parents reported modest increases in 


their confidence in supporting their child’s science learning. 
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Limitations 


Several study design characteristics should be considered when interpreting these findings. 


» This study’s single-group design limits our ability to make claims about causation and 
does not meet What Works Clearinghouse evidence standards (What Works 
Clearinghouse, 2017). 


» The sample is not nationally representative. 


» Wecannot claim that these findings would extend to average users of the Play & Learn 
Science app, because this study included additional supports aligned to the app. 


» The researcher-developed measures are closely aligned with the Play & Learn Science 
app content; we cannot say how participants would perform on less aligned measures, 
such as academic assessments. 


» Wedid not measure inter-assessor agreement in either the assessment or scoring 
process. 


» Small studies, single-group studies, and those with closely aligned measures tend to 
report larger effect sizes than studies with other research designs (Bakker et al., 2019). 


Implications for Practice and Future Research 


Our findings suggest that a digital app can be a catalyst for families’ science exploration and learning, 
when it features rich interactive content and it empowers parents with knowledge and skills to 
effectively facilitate and extend their children’s digital and real-world science experiences. The study 
adds to a body of research that points to young children’s capacity for engaging in science and 
engineering practices and for exploring, thinking about, and talking about science concepts at 
developmentally appropriate levels. This study has the potential to speak directly to families, and to 
the teachers and child-care providers who work with them, as it illustrates how families benefit from 
the types of resources that parents have identified as essential to supporting science engagement in 
prior research studies. Our findings suggest that parents will actively engage in doing and learning 
science with their children when they are provided with a quality app and additional supports for using 


it to the fullest. 
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This is exactly what is taking place in 30 communities across the country as part of the CPB-PBS 
Ready To Learn Initiative Community Collaboratives for Early Learning Media. Children and their 
parents are participating in family science events that provide similar parental supports to those 
included in this study, including guidance for navigating an educational science app, models of the 
hands-on science activities and suggested interactions in the app, and structural supports and 
encouragement for using the app in a connected and sequential manner. Public media’s fundamental 
approach to working with families is to provide these educative supports within nurturing community 
contexts, an approach reflected in this study as the intervention was embedded within two community 
child-care centers. This study suggests that many community-focused organizations, particularly 
those such as child-care centers, schools, and libraries that seek to empower, educate, and engage 
families, could benefit from using the Play & Learn Science app as a model for designing their own 
family-focused science events. The app can support experiences that may be particularly valuable in 
underserved communities with limited access to high-quality science education and science-related 


activities. 
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Appendix A 


Performance-Based Assessment Tasks by PBS KIDS Science Learning Framework Standards 


Performance- 
based task 


Physical science Earth/Space science 


Matter and its interactions: Understand that different kinds of matter exist 
Ramp and Roll: 


Explore the Roll and that they can be described and classified by their observable 


properties. 


Motion and stability; forces and interactions: Understand that pushes and 
pulls can cause objects to move. 
Understand that pushing on an object can change the speed or direction 


of its motion and can start or stop it. 


: Matter and its interactions: Understand that different kinds of matter exist 
Water Games: Sink or 


Bloat and that they can be described and classified by their observable 


properties. 


Motion and stability; forces and interactions: Understand that pushes and 
pulls can cause objects to move. 
Understand that pushes and pulls can have different strengths and 


directions. 


Ready To Learn Research 


PBS KIDS Play & Learn Science Evaluation Report 


Performance- 
based task 


Physical science Earth/Space science 


Shadow Play: Guess Waves and their applications in technologies for information transfer: 


ine Shadow Begin to understand that some materials block all the light creating a 


dark shadow on any surface beyond them, where the light cannot reach. 


Matter and its interactions: Understand that different properties of Earth’s systems: Understand that weather is the 

Gear Up: Gear Up 

materials are suited to different purposes. combination of sunlight, wind, snow, or rain and 
temperature in a particular region at a particular 
time and can be measured by people to describe 
and record the weather and to notice patterns 


over time. 
Matter and its interactions: Understand that different kinds of matter exist 
Gear Up: The 


; and these can be described and classified by their observable 
Amazing Umbrella 


properties. 
Understand that different properties of materials are suited to different 


purposes. 
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Appendix B 


Assessment Scoring 


Vocabulary assessment 

Multiple-choice vocabulary items were scored as correct or incorrect. To identify items exhibiting a 
ceiling effect, we set a threshold of 95% correct responses in the combined pre- and post-experience 
assessment scores. We excluded one item that exceeded this criterion. Scores were computed as 
each child’s average of correct responses on the remaining 18 items, excluding two training items. 
Using averages rather than sums enabled us to count items as skipped (e.g., if the child did not 
complete the assessment) without deducting points. According to our scoring rules, if a child gave 
incorrect responses to five items in a row, the assessor discontinued the assessment and all 
remaining items were scored as zero rather than skipped. Appendix C presents descriptive statistics 


for all items included in the final vocabulary assessment. 


Performance-based tasks 

The performance-based tasks were scored by the two study leads. Scorers established a rubric that 
defined and provided examples of correct and incorrect responses. As a calibration exercise, scorers 
scored six child responses in common, or 10% of the dataset. We used this as an opportunity to 


refine the scoring rubric. Each scorer then scored 50% of the remaining data set. 


As for the vocabulary assessment (described above), we set a ceiling effect threshold of 95% correct 
responses in the combined pre- and post-experience assessment scores and excluded one item that 
exceeded this threshold. Total scores also omit a series of three questions intended to provide a 
second chance to identify an object that both rolls and slides for children who did not initially identify 
one. Only one child correctly identified an object that rolls and slides during the second chance; these 


three items therefore did not contribute meaningful variance to the performance-based task scores. 


As for the vocabulary assessment (described above), scores were computed as the average of 
correct responses for all assessment items. All items were scored out of one point; those initially 
assigned more points for scoring simplicity (e.g., a count of how many objects the child identified) 


were rescaled to one point. 


Vocabulary use during performance-based tasks 

Assessors were told to write down all relevant vocabulary used during the performance-based tasks. 
We extracted child utterances from other assessor notes and then used an automated word search to 
count instances of key vocabulary terms in children’s performance-based task responses (see 
Appendix C). 
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We generated an initial list of vocabulary terms from key terminology used in the Play & Learn 
Science in-app and hands-on activities. We then analyzed the frequency of all words children used 
during the performance-based task and identified additional content-specific terms that appeared 
three or more times in the data set. Appendix C presents the final word set. Alternative endings were 
counted for all terms. 


Parent survey 

Vocabulary words included on the parent survey are listed in Appendix C. This report omits a parent 
post-survey item that asked about the impact of Play & Learn Science games and activities on the 
child’s interest in science activities such as “asking questions about the world around them” and 
“investigating or experimenting to learn about the world.” The response options were reversed relative 
to a similarly structured item directly previous. As a result, responses were overwhelmingly negative. 
We suspect that this was due to the order of the response options and does not reflect parents’ actual 


experiences because it is inconsistent with responses on other similar items. 
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Appendix C 


Item-Level Descriptive Statistics for Vocabulary Measures 


Vocabulary Parent Survey Performance Based AVfoferVo)biEcha's 
Word Assessment Assessment 


Explore the Roll 
Flat 44% 12% 3 3 
Round 12% 88% 2 6 
Steep 13% 28% 0 0 35% 26% 
High 91% 97% 2 0 87% 84% 
Low 66% 81% 1 0 
Ramp 41% 15% 1 0 58% 87% 
Roll 63% 97% 28 38 11% 94% 
Slide 94% 100% 16 14 
Target 38% 59% 52% 71% 
Sink or Float 
Down 22 20 
Up 15 24 
Sink 69% 94% TZ 48 
Float 59% 100% 25 34 68% 81% 
Liquid 41% 50% 
Glass 1 0 
Metal 2 5 
Heavy 97% 97% 13 19 90% 97% 
Guess the Shadow 
Big 8 15 
Small 2 9 71% 81% 
Close 84% 84% 0 4 
Far 69% 81% 0 2 42% 43% 
Dark 94% 91% 3 3 
Light 91% 88% 6 2 
Shadow 53% 88% 9 9 68% 11% 
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Vocabulary Parent Survey Performance Based AVfofer-Vo)biEcha's 
Word Assessment Assessment 


Shape 18% 18% 5 3 
Gear Up 
Weather 18% 88% 5 6 
Cold 100% 97% 18 24 
Cool 81% 91% 0 2 
Warm 90% 97% 9 13 
Hot 97% 97% 19 25 71% 74% 
Sun 100% 100% 15 22 
Rain 100% 97% 36 41 68% 71% 
Snow 18% 88% 4 3 
Wind 18% 94% 2 0 
Temperature 5 10 
Degrees 0 4 
Thermometer 31% 69% 9 16 58% 10% 
Boot 1 5 
Hat 3 5 
Jacket 6 10 
Shirt 0 3 
Shoe 1 2 
Sleeve 3 4 
Umbrella 9 8 
The Amazing Umbrella 
Absorb 9% 31% 0 0 13% 30% 
Repel 0% 6% 0 0 
Soak 31% 59% 2 1 
Liquid 32% 30% 
Fabric 0 0 
Felt 0 0 
Plastic 4 5 
Sponge 1 3 
Wood 0 3 
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Vocabulary Parent Survey Performance Based AVfofer-Vo)biEcha's 
Word Assessment Assessment 


0 2 


Fluffy 
Hard 8 6 
Soft 14 15 
Hole 20 32 
General Terms 
Circle UZ 14 
Oval 3 4 
Square 4 4 
Diamond 3 1 
Side 1 3 
Compare 16% 34% 16% 19% 
Observing 26% 33% 
Long 97% 94% 
Short 84% 88% 
Object 31% 63% 1 3 45% 60% 


Note: Words are grouped by focal topic for display only. Blank cells indicate that a word was not included on an assessment. Words were 
excluded from the performance-based assessment only if they were not spoken by any children. Some measures used alternate word endings 
(e.g., rolling versus roll); terms are collapsed for simplicity. The parent survey specified that alternate endings should be counted in parents’ 
reports. Alternate endings were counted on the performance-based assessment. On the vocabulary assessment only, the assessor spoke the 


terms; there was no need to count child-produced alternate endings. 


Ready To Learn Research 


